NLP Spring 2026 - Homework 2

Textbook Practice

The unnest_tokens function

# load text
text <- c("Because I could not stop for Death -", "He kindly stopped for me -", "The Carriage held but just Ourselves -",
    "and Immortality")
text

## [1] "Because I could not stop for Death -"  
## [2] "He kindly stopped for me -"            
## [3] "The Carriage held but just Ourselves -"
## [4] "and Immortality"

# put text into data frame
library(dplyr)
text_df <- tibble(line = 1:4, text = text)
text_df

line	text
1	Because I could not stop for Death -
2	He kindly stopped for me -
3	The Carriage held but just Ourselves -
4	and Immortality

# tokenize & transform into a tidy data structure - unnest_tokens() function
library(tidytext)
text_df %>%
    unnest_tokens(word, text)

line	word
1	because
1	i
1	could
1	not
1	stop
1	for
1	death
2	he
2	kindly
2	stopped
2	for
2	me
3	the
3	carriage
3	held
3	but
3	just
3	ourselves
4	and
4	immortality

Tidying the works of Jane Austen

library(janeaustenr)
library(dplyr)
library(stringr)

# add columns for row and chapter numbers for lines of text
original_books <- austen_books() %>%
    group_by(book) %>%
    mutate(linenumber = row_number(), chapter = cumsum(str_detect(text, regex("^chapter [\\divxlc]",
        ignore_case = TRUE)))) %>%
    ungroup()

original_books

text	book	linenumber	chapter
SENSE AND SENSIBILITY	Sense & Sensibility	1	0
	Sense & Sensibility	2	0
by Jane Austen	Sense & Sensibility	3	0
	Sense & Sensibility	4	0
	Sense & Sensibility	5	0
	Sense & Sensibility	6	0
	Sense & Sensibility	7	0
	Sense & Sensibility	8	0
	Sense & Sensibility	9	0
CHAPTER 1	Sense & Sensibility	10	1
	Sense & Sensibility	11	1
	Sense & Sensibility	12	1
The family of Dashwood had long been settled in Sussex. Their estate	Sense & Sensibility	13	1
was large, and their residence was at Norland Park, in the centre of	Sense & Sensibility	14	1
their property, where, for many generations, they had lived in so	Sense & Sensibility	15	1
respectable a manner as to engage the general good opinion of their	Sense & Sensibility	16	1
surrounding acquaintance. The late owner of this estate was a single	Sense & Sensibility	17	1
man, who lived to a very advanced age, and who for many years of his	Sense & Sensibility	18	1
life, had a constant companion and housekeeper in his sister. But her	Sense & Sensibility	19	1
death, which happened ten years before his own, produced a great	Sense & Sensibility	20	1
alteration in his home; for to supply her loss, he invited and received	Sense & Sensibility	21	1
into his house the family of his nephew Mr. Henry Dashwood, the legal	Sense & Sensibility	22	1
inheritor of the Norland estate, and the person to whom he intended to	Sense & Sensibility	23	1
bequeath it. In the society of his nephew and niece, and their	Sense & Sensibility	24	1
children, the old Gentleman’s days were comfortably spent. His	Sense & Sensibility	25	1
attachment to them all increased. The constant attention of Mr. and	Sense & Sensibility	26	1
Mrs. Henry Dashwood to his wishes, which proceeded not merely from	Sense & Sensibility	27	1
interest, but from goodness of heart, gave him every degree of solid	Sense & Sensibility	28	1
comfort which his age could receive; and the cheerfulness of the	Sense & Sensibility	29	1
children added a relish to his existence.	Sense & Sensibility	30	1
	Sense & Sensibility	31	1
By a former marriage, Mr. Henry Dashwood had one son: by his present	Sense & Sensibility	32	1
lady, three daughters. The son, a steady respectable young man, was	Sense & Sensibility	33	1
amply provided for by the fortune of his mother, which had been large,	Sense & Sensibility	34	1
and half of which devolved on him on his coming of age. By his own	Sense & Sensibility	35	1
marriage, likewise, which happened soon afterwards, he added to his	Sense & Sensibility	36	1
wealth. To him therefore the succession to the Norland estate was not	Sense & Sensibility	37	1
so really important as to his sisters; for their fortune, independent	Sense & Sensibility	38	1
of what might arise to them from their father’s inheriting that	Sense & Sensibility	39	1
property, could be but small. Their mother had nothing, and their	Sense & Sensibility	40	1
father only seven thousand pounds in his own disposal; for the	Sense & Sensibility	41	1
remaining moiety of his first wife’s fortune was also secured to her	Sense & Sensibility	42	1
child, and he had only a life-interest in it.	Sense & Sensibility	43	1
	Sense & Sensibility	44	1
The old gentleman died: his will was read, and like almost every other	Sense & Sensibility	45	1
will, gave as much disappointment as pleasure. He was neither so	Sense & Sensibility	46	1
unjust, nor so ungrateful, as to leave his estate from his nephew;–but	Sense & Sensibility	47	1
he left it to him on such terms as destroyed half the value of the	Sense & Sensibility	48	1
bequest. Mr. Dashwood had wished for it more for the sake of his wife	Sense & Sensibility	49	1
and daughters than for himself or his son;–but to his son, and his	Sense & Sensibility	50	1
son’s son, a child of four years old, it was secured, in such a way, as	Sense & Sensibility	51	1
to leave to himself no power of providing for those who were most dear	Sense & Sensibility	52	1
to him, and who most needed a provision by any charge on the estate, or	Sense & Sensibility	53	1
by any sale of its valuable woods. The whole was tied up for the	Sense & Sensibility	54	1
benefit of this child, who, in occasional visits with his father and	Sense & Sensibility	55	1
mother at Norland, had so far gained on the affections of his uncle, by	Sense & Sensibility	56	1
such attractions as are by no means unusual in children of two or three	Sense & Sensibility	57	1
years old; an imperfect articulation, an earnest desire of having his	Sense & Sensibility	58	1
own way, many cunning tricks, and a great deal of noise, as to outweigh	Sense & Sensibility	59	1
all the value of all the attention which, for years, he had received	Sense & Sensibility	60	1
from his niece and her daughters. He meant not to be unkind, however,	Sense & Sensibility	61	1
and, as a mark of his affection for the three girls, he left them a	Sense & Sensibility	62	1
thousand pounds a-piece.	Sense & Sensibility	63	1
	Sense & Sensibility	64	1
Mr. Dashwood’s disappointment was, at first, severe; but his temper was	Sense & Sensibility	65	1
cheerful and sanguine; and he might reasonably hope to live many years,	Sense & Sensibility	66	1
and by living economically, lay by a considerable sum from the produce	Sense & Sensibility	67	1
of an estate already large, and capable of almost immediate	Sense & Sensibility	68	1
improvement. But the fortune, which had been so tardy in coming, was	Sense & Sensibility	69	1
his only one twelvemonth. He survived his uncle no longer; and ten	Sense & Sensibility	70	1
thousand pounds, including the late legacies, was all that remained for	Sense & Sensibility	71	1
his widow and daughters.	Sense & Sensibility	72	1
	Sense & Sensibility	73	1
His son was sent for as soon as his danger was known, and to him Mr.	Sense & Sensibility	74	1
Dashwood recommended, with all the strength and urgency which illness	Sense & Sensibility	75	1
could command, the interest of his mother-in-law and sisters.	Sense & Sensibility	76	1
	Sense & Sensibility	77	1
Mr. John Dashwood had not the strong feelings of the rest of the	Sense & Sensibility	78	1
family; but he was affected by a recommendation of such a nature at	Sense & Sensibility	79	1
such a time, and he promised to do every thing in his power to make	Sense & Sensibility	80	1
them comfortable. His father was rendered easy by such an assurance,	Sense & Sensibility	81	1
and Mr. John Dashwood had then leisure to consider how much there might	Sense & Sensibility	82	1
prudently be in his power to do for them.	Sense & Sensibility	83	1
	Sense & Sensibility	84	1
He was not an ill-disposed young man, unless to be rather cold hearted	Sense & Sensibility	85	1
and rather selfish is to be ill-disposed: but he was, in general, well	Sense & Sensibility	86	1
respected; for he conducted himself with propriety in the discharge of	Sense & Sensibility	87	1
his ordinary duties. Had he married a more amiable woman, he might	Sense & Sensibility	88	1
have been made still more respectable than he was:–he might even have	Sense & Sensibility	89	1
been made amiable himself; for he was very young when he married, and	Sense & Sensibility	90	1
very fond of his wife. But Mrs. John Dashwood was a strong caricature	Sense & Sensibility	91	1
of himself;–more narrow-minded and selfish.	Sense & Sensibility	92	1
	Sense & Sensibility	93	1
When he gave his promise to his father, he meditated within himself to	Sense & Sensibility	94	1
increase the fortunes of his sisters by the present of a thousand	Sense & Sensibility	95	1
pounds a-piece. He then really thought himself equal to it. The	Sense & Sensibility	96	1
prospect of four thousand a-year, in addition to his present income,	Sense & Sensibility	97	1
besides the remaining half of his own mother’s fortune, warmed his	Sense & Sensibility	98	1
heart, and made him feel capable of generosity.– “Yes, he would give	Sense & Sensibility	99	1
them three thousand pounds: it would be liberal and handsome! It would	Sense & Sensibility	100	1
be enough to make them completely easy. Three thousand pounds! he	Sense & Sensibility	101	1
could spare so considerable a sum with little inconvenience.”– He	Sense & Sensibility	102	1
thought of it all day long, and for many days successively, and he did	Sense & Sensibility	103	1
not repent.	Sense & Sensibility	104	1
	Sense & Sensibility	105	1
No sooner was his father’s funeral over, than Mrs. John Dashwood,	Sense & Sensibility	106	1
without sending any notice of her intention to her mother-in-law,	Sense & Sensibility	107	1
arrived with her child and their attendants. No one could dispute her	Sense & Sensibility	108	1
right to come; the house was her husband’s from the moment of his	Sense & Sensibility	109	1
father’s decease; but the indelicacy of her conduct was so much the	Sense & Sensibility	110	1
greater, and to a woman in Mrs. Dashwood’s situation, with only common	Sense & Sensibility	111	1
feelings, must have been highly unpleasing;–but in HER mind there was	Sense & Sensibility	112	1
a sense of honor so keen, a generosity so romantic, that any offence of	Sense & Sensibility	113	1
the kind, by whomsoever given or received, was to her a source of	Sense & Sensibility	114	1
immovable disgust. Mrs. John Dashwood had never been a favourite with	Sense & Sensibility	115	1
any of her husband’s family; but she had had no opportunity, till the	Sense & Sensibility	116	1
present, of shewing them with how little attention to the comfort of	Sense & Sensibility	117	1
other people she could act when occasion required it.	Sense & Sensibility	118	1
	Sense & Sensibility	119	1
So acutely did Mrs. Dashwood feel this ungracious behaviour, and so	Sense & Sensibility	120	1
earnestly did she despise her daughter-in-law for it, that, on the	Sense & Sensibility	121	1
arrival of the latter, she would have quitted the house for ever, had	Sense & Sensibility	122	1
not the entreaty of her eldest girl induced her first to reflect on the	Sense & Sensibility	123	1
propriety of going, and her own tender love for all her three children	Sense & Sensibility	124	1
determined her afterwards to stay, and for their sakes avoid a breach	Sense & Sensibility	125	1
with their brother.	Sense & Sensibility	126	1
	Sense & Sensibility	127	1
Elinor, this eldest daughter, whose advice was so effectual, possessed	Sense & Sensibility	128	1
a strength of understanding, and coolness of judgment, which qualified	Sense & Sensibility	129	1
her, though only nineteen, to be the counsellor of her mother, and	Sense & Sensibility	130	1
enabled her frequently to counteract, to the advantage of them all,	Sense & Sensibility	131	1
that eagerness of mind in Mrs. Dashwood which must generally have led	Sense & Sensibility	132	1
to imprudence. She had an excellent heart;–her disposition was	Sense & Sensibility	133	1
affectionate, and her feelings were strong; but she knew how to govern	Sense & Sensibility	134	1
them: it was a knowledge which her mother had yet to learn; and which	Sense & Sensibility	135	1
one of her sisters had resolved never to be taught.	Sense & Sensibility	136	1
	Sense & Sensibility	137	1
Marianne’s abilities were, in many respects, quite equal to Elinor’s.	Sense & Sensibility	138	1
She was sensible and clever; but eager in everything: her sorrows, her	Sense & Sensibility	139	1
joys, could have no moderation. She was generous, amiable,	Sense & Sensibility	140	1
interesting: she was everything but prudent. The resemblance between	Sense & Sensibility	141	1
her and her mother was strikingly great.	Sense & Sensibility	142	1
	Sense & Sensibility	143	1
Elinor saw, with concern, the excess of her sister’s sensibility; but	Sense & Sensibility	144	1
by Mrs. Dashwood it was valued and cherished. They encouraged each	Sense & Sensibility	145	1
other now in the violence of their affliction. The agony of grief	Sense & Sensibility	146	1
which overpowered them at first, was voluntarily renewed, was sought	Sense & Sensibility	147	1
for, was created again and again. They gave themselves up wholly to	Sense & Sensibility	148	1
their sorrow, seeking increase of wretchedness in every reflection that	Sense & Sensibility	149	1
could afford it, and resolved against ever admitting consolation in	Sense & Sensibility	150	1
future. Elinor, too, was deeply afflicted; but still she could	Sense & Sensibility	151	1
struggle, she could exert herself. She could consult with her brother,	Sense & Sensibility	152	1
could receive her sister-in-law on her arrival, and treat her with	Sense & Sensibility	153	1
proper attention; and could strive to rouse her mother to similar	Sense & Sensibility	154	1
exertion, and encourage her to similar forbearance.	Sense & Sensibility	155	1
	Sense & Sensibility	156	1
Margaret, the other sister, was a good-humored, well-disposed girl; but	Sense & Sensibility	157	1
as she had already imbibed a good deal of Marianne’s romance, without	Sense & Sensibility	158	1
having much of her sense, she did not, at thirteen, bid fair to equal	Sense & Sensibility	159	1
her sisters at a more advanced period of life.	Sense & Sensibility	160	1
	Sense & Sensibility	161	1
	Sense & Sensibility	162	1
	Sense & Sensibility	163	1
CHAPTER 2	Sense & Sensibility	164	2
	Sense & Sensibility	165	2
	Sense & Sensibility	166	2
Mrs. John Dashwood now installed herself mistress of Norland; and her	Sense & Sensibility	167	2
mother and sisters-in-law were degraded to the condition of visitors.	Sense & Sensibility	168	2
As such, however, they were treated by her with quiet civility; and by	Sense & Sensibility	169	2
her husband with as much kindness as he could feel towards anybody	Sense & Sensibility	170	2
beyond himself, his wife, and their child. He really pressed them,	Sense & Sensibility	171	2
with some earnestness, to consider Norland as their home; and, as no	Sense & Sensibility	172	2
plan appeared so eligible to Mrs. Dashwood as remaining there till she	Sense & Sensibility	173	2
could accommodate herself with a house in the neighbourhood, his	Sense & Sensibility	174	2
invitation was accepted.	Sense & Sensibility	175	2
	Sense & Sensibility	176	2
A continuance in a place where everything reminded her of former	Sense & Sensibility	177	2
delight, was exactly what suited her mind. In seasons of cheerfulness,	Sense & Sensibility	178	2
no temper could be more cheerful than hers, or possess, in a greater	Sense & Sensibility	179	2
degree, that sanguine expectation of happiness which is happiness	Sense & Sensibility	180	2
itself. But in sorrow she must be equally carried away by her fancy,	Sense & Sensibility	181	2
and as far beyond consolation as in pleasure she was beyond alloy.	Sense & Sensibility	182	2
	Sense & Sensibility	183	2
Mrs. John Dashwood did not at all approve of what her husband intended	Sense & Sensibility	184	2
to do for his sisters. To take three thousand pounds from the fortune	Sense & Sensibility	185	2
of their dear little boy would be impoverishing him to the most	Sense & Sensibility	186	2
dreadful degree. She begged him to think again on the subject. How	Sense & Sensibility	187	2
could he answer it to himself to rob his child, and his only child too,	Sense & Sensibility	188	2
of so large a sum? And what possible claim could the Miss Dashwoods,	Sense & Sensibility	189	2
who were related to him only by half blood, which she considered as no	Sense & Sensibility	190	2
relationship at all, have on his generosity to so large an amount. It	Sense & Sensibility	191	2
was very well known that no affection was ever supposed to exist	Sense & Sensibility	192	2
between the children of any man by different marriages; and why was he	Sense & Sensibility	193	2
to ruin himself, and their poor little Harry, by giving away all his	Sense & Sensibility	194	2
money to his half sisters?	Sense & Sensibility	195	2
	Sense & Sensibility	196	2
“It was my father’s last request to me,” replied her husband, “that I	Sense & Sensibility	197	2
should assist his widow and daughters.”	Sense & Sensibility	198	2
	Sense & Sensibility	199	2
“He did not know what he was talking of, I dare say; ten to one but he	Sense & Sensibility	200	2

# restructure to one-token-per-row - in this case one word per row
library(tidytext)
tidy_books <- original_books %>%
    unnest_tokens(word, text)

tidy_books

book	linenumber	chapter	word
Sense & Sensibility	1	0	sense
Sense & Sensibility	1	0	and
Sense & Sensibility	1	0	sensibility
Sense & Sensibility	3	0	by
Sense & Sensibility	3	0	jane
Sense & Sensibility	3	0	austen
Sense & Sensibility	5	0	1811
Sense & Sensibility	10	1	chapter
Sense & Sensibility	10	1	1
Sense & Sensibility	13	1	the
Sense & Sensibility	13	1	family
Sense & Sensibility	13	1	of
Sense & Sensibility	13	1	dashwood
Sense & Sensibility	13	1	had
Sense & Sensibility	13	1	long
Sense & Sensibility	13	1	been
Sense & Sensibility	13	1	settled
Sense & Sensibility	13	1	in
Sense & Sensibility	13	1	sussex
Sense & Sensibility	13	1	their
Sense & Sensibility	13	1	estate
Sense & Sensibility	14	1	was
Sense & Sensibility	14	1	large
Sense & Sensibility	14	1	and
Sense & Sensibility	14	1	their
Sense & Sensibility	14	1	residence
Sense & Sensibility	14	1	was
Sense & Sensibility	14	1	at
Sense & Sensibility	14	1	norland
Sense & Sensibility	14	1	park
Sense & Sensibility	14	1	in
Sense & Sensibility	14	1	the
Sense & Sensibility	14	1	centre
Sense & Sensibility	14	1	of
Sense & Sensibility	15	1	their
Sense & Sensibility	15	1	property
Sense & Sensibility	15	1	where
Sense & Sensibility	15	1	for
Sense & Sensibility	15	1	many
Sense & Sensibility	15	1	generations
Sense & Sensibility	15	1	they
Sense & Sensibility	15	1	had
Sense & Sensibility	15	1	lived
Sense & Sensibility	15	1	in
Sense & Sensibility	15	1	so
Sense & Sensibility	16	1	respectable
Sense & Sensibility	16	1	a
Sense & Sensibility	16	1	manner
Sense & Sensibility	16	1	as
Sense & Sensibility	16	1	to
Sense & Sensibility	16	1	engage
Sense & Sensibility	16	1	the
Sense & Sensibility	16	1	general
Sense & Sensibility	16	1	good
Sense & Sensibility	16	1	opinion
Sense & Sensibility	16	1	of
Sense & Sensibility	16	1	their
Sense & Sensibility	17	1	surrounding
Sense & Sensibility	17	1	acquaintance
Sense & Sensibility	17	1	the
Sense & Sensibility	17	1	late
Sense & Sensibility	17	1	owner
Sense & Sensibility	17	1	of
Sense & Sensibility	17	1	this
Sense & Sensibility	17	1	estate
Sense & Sensibility	17	1	was
Sense & Sensibility	17	1	a
Sense & Sensibility	17	1	single
Sense & Sensibility	18	1	man
Sense & Sensibility	18	1	who
Sense & Sensibility	18	1	lived
Sense & Sensibility	18	1	to
Sense & Sensibility	18	1	a
Sense & Sensibility	18	1	very
Sense & Sensibility	18	1	advanced
Sense & Sensibility	18	1	age
Sense & Sensibility	18	1	and
Sense & Sensibility	18	1	who
Sense & Sensibility	18	1	for
Sense & Sensibility	18	1	many
Sense & Sensibility	18	1	years
Sense & Sensibility	18	1	of
Sense & Sensibility	18	1	his
Sense & Sensibility	19	1	life
Sense & Sensibility	19	1	had
Sense & Sensibility	19	1	a
Sense & Sensibility	19	1	constant
Sense & Sensibility	19	1	companion
Sense & Sensibility	19	1	and
Sense & Sensibility	19	1	housekeeper
Sense & Sensibility	19	1	in
Sense & Sensibility	19	1	his
Sense & Sensibility	19	1	sister
Sense & Sensibility	19	1	but
Sense & Sensibility	19	1	her
Sense & Sensibility	20	1	death
Sense & Sensibility	20	1	which
Sense & Sensibility	20	1	happened
Sense & Sensibility	20	1	ten
Sense & Sensibility	20	1	years
Sense & Sensibility	20	1	before
Sense & Sensibility	20	1	his
Sense & Sensibility	20	1	own
Sense & Sensibility	20	1	produced
Sense & Sensibility	20	1	a
Sense & Sensibility	20	1	great
Sense & Sensibility	21	1	alteration
Sense & Sensibility	21	1	in
Sense & Sensibility	21	1	his
Sense & Sensibility	21	1	home
Sense & Sensibility	21	1	for
Sense & Sensibility	21	1	to
Sense & Sensibility	21	1	supply
Sense & Sensibility	21	1	her
Sense & Sensibility	21	1	loss
Sense & Sensibility	21	1	he
Sense & Sensibility	21	1	invited
Sense & Sensibility	21	1	and
Sense & Sensibility	21	1	received
Sense & Sensibility	22	1	into
Sense & Sensibility	22	1	his
Sense & Sensibility	22	1	house
Sense & Sensibility	22	1	the
Sense & Sensibility	22	1	family
Sense & Sensibility	22	1	of
Sense & Sensibility	22	1	his
Sense & Sensibility	22	1	nephew
Sense & Sensibility	22	1	mr
Sense & Sensibility	22	1	henry
Sense & Sensibility	22	1	dashwood
Sense & Sensibility	22	1	the
Sense & Sensibility	22	1	legal
Sense & Sensibility	23	1	inheritor
Sense & Sensibility	23	1	of
Sense & Sensibility	23	1	the
Sense & Sensibility	23	1	norland
Sense & Sensibility	23	1	estate
Sense & Sensibility	23	1	and
Sense & Sensibility	23	1	the
Sense & Sensibility	23	1	person
Sense & Sensibility	23	1	to
Sense & Sensibility	23	1	whom
Sense & Sensibility	23	1	he
Sense & Sensibility	23	1	intended
Sense & Sensibility	23	1	to
Sense & Sensibility	24	1	bequeath
Sense & Sensibility	24	1	it
Sense & Sensibility	24	1	in
Sense & Sensibility	24	1	the
Sense & Sensibility	24	1	society
Sense & Sensibility	24	1	of
Sense & Sensibility	24	1	his
Sense & Sensibility	24	1	nephew
Sense & Sensibility	24	1	and
Sense & Sensibility	24	1	niece
Sense & Sensibility	24	1	and
Sense & Sensibility	24	1	their
Sense & Sensibility	25	1	children
Sense & Sensibility	25	1	the
Sense & Sensibility	25	1	old
Sense & Sensibility	25	1	gentleman’s
Sense & Sensibility	25	1	days
Sense & Sensibility	25	1	were
Sense & Sensibility	25	1	comfortably
Sense & Sensibility	25	1	spent
Sense & Sensibility	25	1	his
Sense & Sensibility	26	1	attachment
Sense & Sensibility	26	1	to
Sense & Sensibility	26	1	them
Sense & Sensibility	26	1	all
Sense & Sensibility	26	1	increased
Sense & Sensibility	26	1	the
Sense & Sensibility	26	1	constant
Sense & Sensibility	26	1	attention
Sense & Sensibility	26	1	of
Sense & Sensibility	26	1	mr
Sense & Sensibility	26	1	and
Sense & Sensibility	27	1	mrs
Sense & Sensibility	27	1	henry
Sense & Sensibility	27	1	dashwood
Sense & Sensibility	27	1	to
Sense & Sensibility	27	1	his
Sense & Sensibility	27	1	wishes
Sense & Sensibility	27	1	which
Sense & Sensibility	27	1	proceeded
Sense & Sensibility	27	1	not
Sense & Sensibility	27	1	merely
Sense & Sensibility	27	1	from
Sense & Sensibility	28	1	interest
Sense & Sensibility	28	1	but
Sense & Sensibility	28	1	from
Sense & Sensibility	28	1	goodness
Sense & Sensibility	28	1	of
Sense & Sensibility	28	1	heart
Sense & Sensibility	28	1	gave
Sense & Sensibility	28	1	him
Sense & Sensibility	28	1	every
Sense & Sensibility	28	1	degree
Sense & Sensibility	28	1	of
Sense & Sensibility	28	1	solid

# remove stop words using anti_join() and the tidytext dataset of stop words
data(stop_words)

tidy_books <- tidy_books %>%
    anti_join(stop_words)

tidy_books

book	linenumber	chapter	word
Sense & Sensibility	1	0	sense
Sense & Sensibility	1	0	sensibility
Sense & Sensibility	3	0	jane
Sense & Sensibility	3	0	austen
Sense & Sensibility	5	0	1811
Sense & Sensibility	10	1	chapter
Sense & Sensibility	10	1	1
Sense & Sensibility	13	1	family
Sense & Sensibility	13	1	dashwood
Sense & Sensibility	13	1	settled
Sense & Sensibility	13	1	sussex
Sense & Sensibility	13	1	estate
Sense & Sensibility	14	1	residence
Sense & Sensibility	14	1	norland
Sense & Sensibility	14	1	park
Sense & Sensibility	14	1	centre
Sense & Sensibility	15	1	property
Sense & Sensibility	15	1	generations
Sense & Sensibility	15	1	lived
Sense & Sensibility	16	1	respectable
Sense & Sensibility	16	1	manner
Sense & Sensibility	16	1	engage
Sense & Sensibility	16	1	opinion
Sense & Sensibility	17	1	surrounding
Sense & Sensibility	17	1	acquaintance
Sense & Sensibility	17	1	late
Sense & Sensibility	17	1	owner
Sense & Sensibility	17	1	estate
Sense & Sensibility	17	1	single
Sense & Sensibility	18	1	lived
Sense & Sensibility	18	1	advanced
Sense & Sensibility	18	1	age
Sense & Sensibility	19	1	life
Sense & Sensibility	19	1	constant
Sense & Sensibility	19	1	companion
Sense & Sensibility	19	1	housekeeper
Sense & Sensibility	19	1	sister
Sense & Sensibility	20	1	death
Sense & Sensibility	20	1	happened
Sense & Sensibility	20	1	ten
Sense & Sensibility	20	1	produced
Sense & Sensibility	21	1	alteration
Sense & Sensibility	21	1	home
Sense & Sensibility	21	1	supply
Sense & Sensibility	21	1	loss
Sense & Sensibility	21	1	invited
Sense & Sensibility	21	1	received
Sense & Sensibility	22	1	house
Sense & Sensibility	22	1	family
Sense & Sensibility	22	1	nephew
Sense & Sensibility	22	1	henry
Sense & Sensibility	22	1	dashwood
Sense & Sensibility	22	1	legal
Sense & Sensibility	23	1	inheritor
Sense & Sensibility	23	1	norland
Sense & Sensibility	23	1	estate
Sense & Sensibility	23	1	person
Sense & Sensibility	23	1	intended
Sense & Sensibility	24	1	bequeath
Sense & Sensibility	24	1	society
Sense & Sensibility	24	1	nephew
Sense & Sensibility	24	1	niece
Sense & Sensibility	25	1	children
Sense & Sensibility	25	1	gentleman’s
Sense & Sensibility	25	1	days
Sense & Sensibility	25	1	comfortably
Sense & Sensibility	25	1	spent
Sense & Sensibility	26	1	attachment
Sense & Sensibility	26	1	increased
Sense & Sensibility	26	1	constant
Sense & Sensibility	26	1	attention
Sense & Sensibility	27	1	henry
Sense & Sensibility	27	1	dashwood
Sense & Sensibility	27	1	wishes
Sense & Sensibility	27	1	proceeded
Sense & Sensibility	28	1	goodness
Sense & Sensibility	28	1	heart
Sense & Sensibility	28	1	degree
Sense & Sensibility	28	1	solid
Sense & Sensibility	29	1	comfort
Sense & Sensibility	29	1	age
Sense & Sensibility	29	1	receive
Sense & Sensibility	29	1	cheerfulness
Sense & Sensibility	30	1	children
Sense & Sensibility	30	1	added
Sense & Sensibility	30	1	relish
Sense & Sensibility	30	1	existence
Sense & Sensibility	32	1	marriage
Sense & Sensibility	32	1	henry
Sense & Sensibility	32	1	dashwood
Sense & Sensibility	32	1	son
Sense & Sensibility	33	1	lady
Sense & Sensibility	33	1	daughters
Sense & Sensibility	33	1	son
Sense & Sensibility	33	1	steady
Sense & Sensibility	33	1	respectable
Sense & Sensibility	34	1	amply
Sense & Sensibility	34	1	provided
Sense & Sensibility	34	1	fortune
Sense & Sensibility	34	1	mother
Sense & Sensibility	35	1	half
Sense & Sensibility	35	1	devolved
Sense & Sensibility	35	1	coming
Sense & Sensibility	35	1	age
Sense & Sensibility	36	1	marriage
Sense & Sensibility	36	1	likewise
Sense & Sensibility	36	1	happened
Sense & Sensibility	36	1	added
Sense & Sensibility	37	1	wealth
Sense & Sensibility	37	1	succession
Sense & Sensibility	37	1	norland
Sense & Sensibility	37	1	estate
Sense & Sensibility	38	1	sisters
Sense & Sensibility	38	1	fortune
Sense & Sensibility	38	1	independent
Sense & Sensibility	39	1	arise
Sense & Sensibility	39	1	father’s
Sense & Sensibility	39	1	inheriting
Sense & Sensibility	40	1	property
Sense & Sensibility	40	1	mother
Sense & Sensibility	41	1	father
Sense & Sensibility	41	1	thousand
Sense & Sensibility	41	1	pounds
Sense & Sensibility	41	1	disposal
Sense & Sensibility	42	1	remaining
Sense & Sensibility	42	1	moiety
Sense & Sensibility	42	1	wife’s
Sense & Sensibility	42	1	fortune
Sense & Sensibility	42	1	secured
Sense & Sensibility	43	1	child
Sense & Sensibility	43	1	life
Sense & Sensibility	45	1	gentleman
Sense & Sensibility	45	1	died
Sense & Sensibility	45	1	read
Sense & Sensibility	46	1	disappointment
Sense & Sensibility	46	1	pleasure
Sense & Sensibility	47	1	unjust
Sense & Sensibility	47	1	ungrateful
Sense & Sensibility	47	1	leave
Sense & Sensibility	47	1	estate
Sense & Sensibility	47	1	nephew
Sense & Sensibility	48	1	left
Sense & Sensibility	48	1	terms
Sense & Sensibility	48	1	destroyed
Sense & Sensibility	48	1	half
Sense & Sensibility	49	1	bequest
Sense & Sensibility	49	1	dashwood
Sense & Sensibility	49	1	wished
Sense & Sensibility	49	1	sake
Sense & Sensibility	49	1	wife
Sense & Sensibility	50	1	daughters
Sense & Sensibility	50	1	son
Sense & Sensibility	50	1	son
Sense & Sensibility	51	1	son’s
Sense & Sensibility	51	1	son
Sense & Sensibility	51	1	child
Sense & Sensibility	51	1	secured
Sense & Sensibility	52	1	leave
Sense & Sensibility	52	1	power
Sense & Sensibility	52	1	providing
Sense & Sensibility	52	1	dear
Sense & Sensibility	53	1	provision
Sense & Sensibility	53	1	charge
Sense & Sensibility	53	1	estate
Sense & Sensibility	54	1	sale
Sense & Sensibility	54	1	valuable
Sense & Sensibility	54	1	woods
Sense & Sensibility	54	1	tied
Sense & Sensibility	55	1	benefit
Sense & Sensibility	55	1	child
Sense & Sensibility	55	1	occasional
Sense & Sensibility	55	1	visits
Sense & Sensibility	55	1	father
Sense & Sensibility	56	1	mother
Sense & Sensibility	56	1	norland
Sense & Sensibility	56	1	gained
Sense & Sensibility	56	1	affections
Sense & Sensibility	56	1	uncle
Sense & Sensibility	57	1	attractions
Sense & Sensibility	57	1	means
Sense & Sensibility	57	1	unusual
Sense & Sensibility	57	1	children
Sense & Sensibility	58	1	imperfect
Sense & Sensibility	58	1	articulation
Sense & Sensibility	58	1	earnest
Sense & Sensibility	58	1	desire
Sense & Sensibility	59	1	cunning
Sense & Sensibility	59	1	tricks
Sense & Sensibility	59	1	deal
Sense & Sensibility	59	1	noise
Sense & Sensibility	59	1	outweigh
Sense & Sensibility	60	1	attention
Sense & Sensibility	60	1	received
Sense & Sensibility	61	1	niece
Sense & Sensibility	61	1	daughters
Sense & Sensibility	61	1	meant
Sense & Sensibility	61	1	unkind
Sense & Sensibility	62	1	mark
Sense & Sensibility	62	1	affection
Sense & Sensibility	62	1	girls

# use count() function to find the most common words in all the books as a
# whole
tidy_books %>%
    count(word, sort = TRUE)

word	n
miss	1855
time	1337
fanny	862
dear	822
lady	817
sir	806
day	797
emma	787
sister	727
house	699
elizabeth	687
elinor	623
hope	601
friend	593
family	578
mind	563
father	558
home	550
jane	550
mother	541
catherine	539
feelings	535
happy	534
moment	515
half	501
love	495
till	495
crawford	493
marianne	492
heart	470
found	469
heard	468
anne	467
pleasure	462
morning	455
letter	450
poor	424
harriet	419
subject	418
brother	416
woman	416
world	410
left	407
cried	405
looked	404
feel	400
speak	395
evening	392
hear	392
weston	389
replied	387
manner	382
darcy	373
happiness	369
edmund	364
party	359
knightley	356
people	352
life	347
captain	342
told	339
opinion	337
spirits	334
suppose	329
acquaintance	328
immediately	323
friends	320
elton	319
ill	317
short	316
passed	314
leave	309
hour	308
idea	304
deal	303
eyes	302
word	298
attention	297
bennet	294
thomas	294
visit	294
colonel	292
comfort	292
coming	292
sort	292
return	287
brought	283
john	282
doubt	281
obliged	281
rest	281
answer	277
woodhouse	277
character	275
affection	272
chapter	272
minutes	272
perfectly	271
walk	264
aunt	263
glad	263
account	259
person	259
bingley	257
elliot	254
means	249
set	249
feeling	247
business	246
situation	245
wife	245
wished	245
hand	244
henry	244
days	243
talked	243
agreeable	242
pretty	242
added	241
town	241
bertram	240
daughter	240
received	240
talk	239
door	237
ladies	235
walked	235
carriage	233
conversation	233
object	233
body	232
dashwood	231
understand	231
manners	230
mary	229
spoke	229
reason	228
continued	227
called	226
ready	226
edward	225
head	223
london	223
returned	221
voice	220
bath	219
dare	219
determined	216
fine	216
sisters	216
impossible	215
marriage	214
regard	214
settled	213
married	211
read	211
fairfax	210
girl	210
care	209
frank	209
power	209
words	209
children	208
nature	208
change	204
expected	201
sense	201
country	200
kindness	200
jennings	199
talking	199
afraid	198
fortune	197
matter	197
appeared	196
giving	196
satisfied	196
thinking	196
tilney	196
directly	195
marry	195
stay	194
william	194
churchill	193
engaged	193
equal	193
assure	192
met	192
night	192
son	191
wentworth	191
believed	188
true	188
uncle	188
isabella	187
dinner	186
pleased	186
norris	185
rushworth	185
usual	185

# create visualization of most common words using ggplot2
library(ggplot2)

tidy_books %>%
    count(word, sort = TRUE) %>%
    filter(n > 600) %>%
    mutate(word = reorder(word, n)) %>%
    ggplot(aes(n, word)) + geom_col() + labs(y = NULL)

## The gutenbergr package

library(gutenbergr)

# Get text for The Time Machine, The War of the Worlds, The Invisible Man, and
# The Island of Doctor Moreau
hgwells <- gutenberg_download(c(35, 36, 5230, 159))

# Tokenize by word and remove stop words
tidy_hgwells <- hgwells %>%
    unnest_tokens(word, text) %>%
    anti_join(stop_words)

# Determine most common words using count() function
tidy_hgwells %>%
    count(word, sort = TRUE)

word	n
time	461
people	302
door	260
heard	249
black	232
stood	229
white	224
hand	218
kemp	213
eyes	210
suddenly	210
found	200
invisible	200
night	200
day	189
montgomery	179
round	178
house	173
it’s	171
light	171
moment	171
head	166
hall	165
martians	165
looked	161
voice	155
strange	149
feet	147
left	147
mind	147
red	140
road	140
water	136
machine	133
presently	132
window	129
half	125
moreau	125
lay	123
human	118
hands	117
sat	116
air	115
trees	115
beast	114
marvel	114
don’t	113
fire	110
sky	108
hill	107
dark	106
running	105
green	104
passed	102
smoke	100
dead	99
i’m	99
coming	96
ran	96
brother	95
slowly	95
grew	93
stopped	92
world	92
days	91
fell	90
pit	90
street	90
told	87
sound	85
darkness	84
morning	84
body	83
ground	83
stared	83
staring	83
grey	81
london	81
houses	80
sea	80
martian	79
life	78
corner	77
common	76
fear	76
beach	75
creatures	75
glass	74
heat	74
idea	74
space	74
struck	74
appeared	73
blue	73
creature	73
story	73
sun	73
table	73
law	72
standing	72
blood	71
arm	70
earth	70
rose	70
cried	68
happened	68
pain	68
past	68
save	68
cylinder	67
food	67
island	67
close	66
minute	66
stranger	66
faint	65
empty	64
silent	64
remember	63
vanished	63
death	62
forward	62
huge	61
animal	60
crowd	60
he’s	60
scarcely	60
set	60
shoulder	60
that’s	60
understand	60
broken	59
caught	59
doubt	59
dog	58
figure	58
books	57
i’ve	57
sight	57
traveller	57
altogether	56
front	56
revolver	54
guns	53
heavy	53
hundred	53
mouth	53
woking	53
adye	52
curious	52
paper	52
watching	52
bright	51
can’t	51
foot	51
line	51
bar	50
escape	50
suppose	50
edge	49
eye	49
feeling	49
hot	49
remained	49
rushed	49
sudden	49
abruptly	48
bushes	48
dust	48
flung	48
imagine	48
matter	48
morlocks	48
nearer	48
terror	48
there’s	48
walked	48
bit	47
kitchen	47
rest	47
silence	47
station	47
direction	46
odd	46
smashed	46
weena	46
cut	45
hair	45
hard	45
sir	45
yards	45
afternoon	44
growing	44
headlong	44
lit	44
noise	44
quiet	44
shouting	44
bunting	43
country	43

# Get text for Jane Eyre, Wuthering Heights, The Tenant of Wildfell Hall,
# Villette, and Agnes Grey
bronte <- gutenberg_download(c(1260, 768, 969, 9182, 767))

tidy_bronte <- bronte %>%
    unnest_tokens(word, text) %>%
    anti_join(stop_words)

# Determine most common wrods using count() function
tidy_bronte %>%
    count(word, sort = TRUE)

word	n
time	1065
miss	854
day	825
don’t	780
hand	767
eyes	714
night	648
heart	638
looked	601
door	591
till	584
half	583
house	580
love	565
life	560
heard	510
left	492
mind	492
sir	490
head	487
home	461
heathcliff	421
mother	417
moment	409
found	405
morning	405
leave	404
told	404
lady	403
i’ll	399
round	386
stood	374
i’m	366
replied	365
john	364
evening	362
jane	362
words	357
speak	356
answered	355
cried	349
hear	348
light	348
hour	346
linton	346
rose	344
voice	344
madame	343
child	339
catherine	338
hope	331
master	331
eye	330
sat	329
fire	328
word	318
rochester	317
father	316
god	309
feel	306
hands	305
mine	303
answer	300
graham	299
cold	294
world	293
poor	290
continued	288
dark	285
suppose	284
brought	280
passed	279
bed	275
pleasure	271
fear	270
care	268
woman	268
read	262
talk	262
return	259
strange	259
called	256
window	256
girl	255
quiet	255
friends	254
white	253
set	252
helen	251
days	245
rest	242
table	242
lay	241
papa	240
nature	239
smile	238
friend	237
returned	236
coming	235
matter	235
dr	234
happy	233
chair	232
dear	231
walk	231
entered	229
book	227
black	226
bretton	225
feelings	223
minutes	218
close	217
live	216
chapter	215
hall	213
remember	213
it’s	212
air	211
de	210
doubt	210
glad	210
huntingdon	209
deep	206
manner	206
people	206
presence	206
wished	206
hair	204
school	204
sense	202
strong	202
lips	201
sort	201
spoke	201
garden	200
fine	199
sister	198
call	197
feeling	197
true	197
ladies	196
stay	195
arthur	194
can’t	194
wife	194
death	193
sit	193
short	191
won’t	191
appeared	190
letter	190
truth	190
children	189
held	189
lucy	189
bear	188
idea	188
spirit	187
i’ve	186
son	184
person	183
st	183
silence	182
trouble	181
exclaimed	179
grey	177
low	177
business	176
sweet	175
company	174
added	172
mary	172
hard	171
power	171
question	171
tone	171
arm	170
times	170
hareton	169
mamma	167
paul	167
brother	166
ill	165
reason	165
character	162
fell	162
pale	162
pretty	162
arms	160
bright	160
he’s	160
past	160
suddenly	160
drawing	159
gentleman	159
step	159
struck	159
subject	159
water	159
bad	158

# Bind data frames for Jane Austen, Bronte sisters, and H.G. Wells
library(tidyr)

frequency <- bind_rows(mutate(tidy_bronte, author = "Brontë Sisters"), mutate(tidy_hgwells,
    author = "H.G. Wells"), mutate(tidy_books, author = "Jane Austen")) %>%
    mutate(word = str_extract(word, "[a-z']+")) %>%
    count(author, word) %>%
    group_by(author) %>%
    mutate(proportion = n/sum(n)) %>%
    select(-n) %>%
    pivot_wider(names_from = author, values_from = proportion) %>%
    pivot_longer(`Brontë Sisters`:`H.G. Wells`, names_to = "author", values_to = "proportion")

frequency

word	Jane Austen	author	proportion
a	0	Brontë Sisters	0
a	0	H.G. Wells	0
aback	NA	Brontë Sisters	0
aback	NA	H.G. Wells	0
abaht	NA	Brontë Sisters	0
abaht	NA	H.G. Wells	NA
abandon	NA	Brontë Sisters	0
abandon	NA	H.G. Wells	0
abandoned	0	Brontë Sisters	0
abandoned	0	H.G. Wells	0
abandoning	NA	Brontë Sisters	0
abandoning	NA	H.G. Wells	0
abandonment	NA	Brontë Sisters	0
abandonment	NA	H.G. Wells	0
abase	NA	Brontë Sisters	0
abase	NA	H.G. Wells	NA
abasement	NA	Brontë Sisters	0
abasement	NA	H.G. Wells	NA
abashed	0	Brontë Sisters	0
abashed	0	H.G. Wells	NA
abate	0	Brontë Sisters	0
abate	0	H.G. Wells	NA
abated	NA	Brontë Sisters	0
abated	NA	H.G. Wells	NA
abating	0	Brontë Sisters	0
abating	0	H.G. Wells	NA
abbot	NA	Brontë Sisters	0
abbot	NA	H.G. Wells	NA
abbreviation	0	Brontë Sisters	0
abbreviation	0	H.G. Wells	NA
abc	NA	Brontë Sisters	0
abc	NA	H.G. Wells	NA
abdicated	NA	Brontë Sisters	0
abdicated	NA	H.G. Wells	NA
abduction	NA	Brontë Sisters	0
abduction	NA	H.G. Wells	NA
abed	NA	Brontë Sisters	0
abed	NA	H.G. Wells	NA
aberdeen	NA	Brontë Sisters	0
aberdeen	NA	H.G. Wells	NA
aberration	NA	Brontë Sisters	0
aberration	NA	H.G. Wells	NA
abetted	NA	Brontë Sisters	0
abetted	NA	H.G. Wells	NA
abhor	0	Brontë Sisters	0
abhor	0	H.G. Wells	NA
abhorred	0	Brontë Sisters	0
abhorred	0	H.G. Wells	NA
abhorrence	0	Brontë Sisters	0
abhorrence	0	H.G. Wells	0
abhors	NA	Brontë Sisters	0
abhors	NA	H.G. Wells	NA
abide	0	Brontë Sisters	0
abide	0	H.G. Wells	0
abigail	NA	Brontë Sisters	0
abigail	NA	H.G. Wells	NA
abigails	NA	Brontë Sisters	0
abigails	NA	H.G. Wells	NA
abilities	0	Brontë Sisters	0
abilities	0	H.G. Wells	NA
ability	0	Brontë Sisters	0
ability	0	H.G. Wells	0
abject	NA	Brontë Sisters	0
abject	NA	H.G. Wells	0
abjured	NA	Brontë Sisters	0
abjured	NA	H.G. Wells	NA
ablaze	NA	Brontë Sisters	0
ablaze	NA	H.G. Wells	0
ablutions	NA	Brontë Sisters	0
ablutions	NA	H.G. Wells	NA
ably	NA	Brontë Sisters	0
ably	NA	H.G. Wells	NA
abnormal	NA	Brontë Sisters	0
abnormal	NA	H.G. Wells	0
abode	0	Brontë Sisters	0
abode	0	H.G. Wells	NA
abodes	NA	Brontë Sisters	0
abodes	NA	H.G. Wells	NA
abominable	0	Brontë Sisters	0
abominable	0	H.G. Wells	0
abominably	0	Brontë Sisters	0
abominably	0	H.G. Wells	NA
aboon	NA	Brontë Sisters	0
aboon	NA	H.G. Wells	NA
aboriginal	NA	Brontë Sisters	0
aboriginal	NA	H.G. Wells	NA
abortive	NA	Brontë Sisters	0
abortive	NA	H.G. Wells	NA
abounded	0	Brontë Sisters	0
abounded	0	H.G. Wells	0
abounding	0	Brontë Sisters	0
abounding	0	H.G. Wells	NA
abridge	0	Brontë Sisters	0
abridge	0	H.G. Wells	NA
abridged	NA	Brontë Sisters	0
abridged	NA	H.G. Wells	NA
abridgment	NA	Brontë Sisters	0
abridgment	NA	H.G. Wells	NA
abroad	0	Brontë Sisters	0
abroad	0	H.G. Wells	0
abrupt	0	Brontë Sisters	0
abrupt	0	H.G. Wells	0
abruptly	0	Brontë Sisters	0
abruptly	0	H.G. Wells	0
abruptness	0	Brontë Sisters	0
abruptness	0	H.G. Wells	0
absalom	NA	Brontë Sisters	0
absalom	NA	H.G. Wells	NA
absconded	NA	Brontë Sisters	0
absconded	NA	H.G. Wells	NA
absence	0	Brontë Sisters	0
absence	0	H.G. Wells	0
absences	0	Brontë Sisters	0
absences	0	H.G. Wells	NA
absent	0	Brontë Sisters	0
absent	0	H.G. Wells	0
absented	0	Brontë Sisters	0
absented	0	H.G. Wells	NA
absently	NA	Brontë Sisters	0
absently	NA	H.G. Wells	NA
absolute	0	Brontë Sisters	0
absolute	0	H.G. Wells	0
absolutely	0	Brontë Sisters	0
absolutely	0	H.G. Wells	0
absolutism	NA	Brontë Sisters	0
absolutism	NA	H.G. Wells	NA
absolve	NA	Brontë Sisters	0
absolve	NA	H.G. Wells	NA
absolved	NA	Brontë Sisters	0
absolved	NA	H.G. Wells	NA
absolves	NA	Brontë Sisters	0
absolves	NA	H.G. Wells	NA
absorb	NA	Brontë Sisters	0
absorb	NA	H.G. Wells	0
absorbed	0	Brontë Sisters	0
absorbed	0	H.G. Wells	0
absorbing	0	Brontë Sisters	0
absorbing	0	H.G. Wells	NA
abstain	0	Brontë Sisters	0
abstain	0	H.G. Wells	0
abstained	0	Brontë Sisters	0
abstained	0	H.G. Wells	NA
abstaining	NA	Brontë Sisters	0
abstaining	NA	H.G. Wells	NA
abstemious	NA	Brontë Sisters	0
abstemious	NA	H.G. Wells	NA
abstemiousness	NA	Brontë Sisters	0
abstemiousness	NA	H.G. Wells	NA
abstinence	0	Brontë Sisters	0
abstinence	0	H.G. Wells	NA
abstract	0	Brontë Sisters	0
abstract	0	H.G. Wells	0
abstracted	0	Brontë Sisters	0
abstracted	0	H.G. Wells	NA
abstractedly	NA	Brontë Sisters	0
abstractedly	NA	H.G. Wells	NA
abstracting	NA	Brontë Sisters	0
abstracting	NA	H.G. Wells	NA
abstraction	0	Brontë Sisters	0
abstraction	0	H.G. Wells	0
abstractions	NA	Brontë Sisters	0
abstractions	NA	H.G. Wells	0
absurd	0	Brontë Sisters	0
absurd	0	H.G. Wells	0
absurdities	0	Brontë Sisters	0
absurdities	0	H.G. Wells	NA
absurdity	0	Brontë Sisters	0
absurdity	0	H.G. Wells	0
absurdly	NA	Brontë Sisters	0
absurdly	NA	H.G. Wells	0
abundance	0	Brontë Sisters	0
abundance	0	H.G. Wells	0
abundant	0	Brontë Sisters	0
abundant	0	H.G. Wells	0
abundantly	0	Brontë Sisters	0
abundantly	0	H.G. Wells	NA
abuse	0	Brontë Sisters	0
abuse	0	H.G. Wells	0
abused	0	Brontë Sisters	0
abused	0	H.G. Wells	0
abuses	0	Brontë Sisters	0
abuses	0	H.G. Wells	NA
abusing	0	Brontë Sisters	0
abusing	0	H.G. Wells	NA
abusive	0	Brontë Sisters	0
abusive	0	H.G. Wells	NA
abyss	0	Brontë Sisters	0
abyss	0	H.G. Wells	NA
acacia	0	Brontë Sisters	0
acacia	0	H.G. Wells	NA
academician	NA	Brontë Sisters	0
academician	NA	H.G. Wells	NA
academicians	NA	Brontë Sisters	0
academicians	NA	H.G. Wells	NA
accede	0	Brontë Sisters	0
accede	0	H.G. Wells	NA
acceded	0	Brontë Sisters	0
acceded	0	H.G. Wells	NA
accelerated	NA	Brontë Sisters	0
accelerated	NA	H.G. Wells	0

# plot findings
library(scales)

ggplot(frequency, aes(x = proportion, y = `Jane Austen`, color = abs(`Jane Austen` -
    proportion))) + geom_abline(color = "gray40", lty = 2) + geom_jitter(alpha = 0.1,
    size = 2.5, width = 0.3, height = 0.3) + geom_text(aes(label = word), check_overlap = TRUE,
    vjust = 1.5) + scale_x_log10(labels = percent_format()) + scale_y_log10(labels = percent_format()) +
    scale_color_gradient(limits = c(0, 0.001), low = "darkslategray4", high = "gray75") +
    facet_wrap(~author, ncol = 2) + theme(legend.position = "none") + labs(y = "Jane Austen",
    x = NULL)

# correlation test to determine how similar and different sets of word
# frequencies are
cor.test(data = frequency[frequency$author == "Brontë Sisters", ], ~proportion +
    `Jane Austen`)

## 
##  Pearson's product-moment correlation
## 
## data:  proportion and Jane Austen
## t = 110.73, df = 10275, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  0.7286645 0.7462983
## sample estimates:
##       cor 
## 0.7376071

cor.test(data = frequency[frequency$author == "H.G. Wells", ], ~proportion + `Jane Austen`)

## 
##  Pearson's product-moment correlation
## 
## data:  proportion and Jane Austen
## t = 35.229, df = 6008, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  0.3925925 0.4345058
## sample estimates:
##       cor 
## 0.4137684

In-Class Practice

My First Function - I will create a function that adds two numbers.

# create function to add two numbers
myfunc <- function(x, y) {
    z <- x + y
    return(z)
}

# run function to add 1 + 9
myfunc(1, 9)

## [1] 10

Update My First Function - Now I will update the function to take the mean of two numbers.

# change function to compute the mean between two numbers
myfunc <- function(x, y, v) {
    z <- (x + y)/v
    return(z)
}

# run function to take average of 1 and 9
myfunc(1, 9, 2)

## [1] 5

# try again
myfunc_revised <- function(...) {
    args <- list(...)
    z <- sum(...)/length(args)
    return(z)
}

myfunc_revised(1, 9)

## [1] 5

myfunc_revised(1, 1, 7)

## [1] 3

In-Class Challenge!

Word Removal

I am going to create a function to remove a word from a string. Then, I will test the function by trying to remove the word “Mary” from “Mary had a little lamb”

# create function to remove 'word' from 'mystring' and return the updated
# string
remove_word_func <- function(mystring, word) {
    x <- mystring
    z <- word
    y <- gsub(z, "", x)
    return(y)
}

# run function to remove 'Mary' from 'Mary had a little lamb'
remove_word_func("Mary had a little lamb", "Mary")

## [1] " had a little lamb"

Punctuation Removal

Now, I am going to create a function that removes punctuation from a string. Then, I will test the function by trying to remove “?!?!” from “Mary had a little lamb?!?!”

# create function to remove punctuation from 'mystring' and return the updated
# string
remove_punc_func <- function(mystring) {
    x <- mystring
    y <- gsub("[[:punct:]]", "", x)
    return(y)
}

# run function to remove '?!?!' from 'Mary had a little lamb?!?!'
remove_punc_func("Mary had a little lamb?!?!")

## [1] "Mary had a little lamb"

Level Up

First I am going to remove non-word tokens (aka punctuation) from a string. Then I am going to split the string to one word per row.

Remove non-word tokens using previous remove_punc_func function

# use remove_punc_func to remove punctuation from 'Mary had a little lamb?!?!')
# and call this new string marydat
marydat <- remove_punc_func("Mary had a little lamb?!?!")
marydat

## [1] "Mary had a little lamb"

Split string to one word per row

# create data frame that takes marydat string and splits into one word per row
df <- data.frame(word = unlist(strsplit(marydat, "\\s+")))

# print data frame
print(df)

##     word
## 1   Mary
## 2    had
## 3      a
## 4 little
## 5   lamb

# try using textbook method
mary_df <- tibble(text = marydat)
mary_df %>%
    unnest_tokens(word, text)

word
mary
had
a
little
lamb

Additional Class Stuff

Importing Text into R

Plain text (txt) and csv files are your go-to formats. There are times when you will want to retain metadata (document titles, dates, etc.) for multidocument corpora.

Local Txt File

From your machine’s ‘data’ folder inside of this project folder. The problem with this is of course that another user can’t knit your markdown unless they install

mytext1 <- readLines("data/pangram.txt")
print(mytext1)

## [1] "The quick brown fox jumped over the lazy dog."

Local CSV File

CSV file with some meta-data appended

mytext2 <- read.csv("data/pantext.csv")
print(mytext2)

##      doc_id
## 1 MySample1
##                                                                               doc_text
## 1 The quick brown fox jumps over the lazy dog. Jack and Jill went up hill. I see cats.

URL Public File How To

Store your text file in a public repository (e.g., your github repo). Make sure that you have the rights to share the text. Copy the URL for the raw file.

# read txt file from github, paste all lines from text into one continuous file
# using ' ' as separator
sherlock <- paste(readLines("https://raw.githubusercontent.com/Reilly-ConceptsCognitionLab/reillylab_publicdata/main/SherlockHolmes.txt"))

# Print the first few lines to check content
cat(sherlock[1:5], sep = "\n")

## To Sherlock Holmes she is always _the_ woman. I have seldom heard him
## mention her under any other name. In his eyes she eclipses and
## predominates the whole of her sex. It was not that he felt any emotion
## akin to love for Irene Adler. All emotions, and that one particularly,
## were abhorrent to his cold, precise but admirably balanced mind. He

Import Unabomber

Import and inspect the raw unabomber manifesto, inspect

unabomb <- paste(readLines("https://raw.githubusercontent.com/Reilly-ConceptsCognitionLab/reillylab_publicdata/main/unabomber_manifesto.txt"))

# Print first few lines of unabomber manifesto
cat(unabomb[1:10], sep = "\n")

## INDUSTRIAL SOCIETY AND ITS FUTURE
## 
## Introduction
## 
## 1. The Industrial Revolution and its consequences have been a disaster for the 
##    human race. They have greatly increased the life-expectancy of those of us 
##    who live in “advanced” countries, but they have destabilized society, have 
##    made life unfulfilling, have subjected human beings to indignities, have led 
##    to widespread psychological suffering (in the Third World to physical 
##    suffering as well) and have inflicted severe damage on the natural world. The

Exporting from R

Sometimes you want to save a text sample to your own machine. You can save it in raw plain text format or RDA format (a compressed format that R likes and will keep you under file size limits)

# create a new dataframe
mysample <- c("Mary had a little lamb")
print(mysample)

## [1] "Mary had a little lamb"

# write that new dataframe to your hard drive as a CSV file
write.csv(mysample, "scratch/mysample.csv", row.names = F)

# Saving text in a compressed format R likes
save(mysample, file = "scratch/mysample.rda")

Importing BIG data

We’ll look at the Jane Austen package and then mess with Project Gutenberg

# if regular installation doesn't work try this: install.packages('gutenbergr',
# type = 'source')

# Load libraries
library(gutenbergr)
library(janeaustenr)

gutenbergr

visit project gutenberg
Project Gutenberg contains >60,000 free works in many different formats, languages, and file types. What we want for text mining is plain text. More specifically, we want structured text files. Files don’t come automatically prepped that way from Project Gutenberg, but we can structure them ourselves. Per the Gutenberg website:

“Project Gutenberg was the first provider of free electronic books, or eBooks. Michael Hart, founder of Project Gutenberg, invented eBooks in 1971 and his memory continues to inspire the creation of eBooks and related content today.”– Return specific criteria:

gutenberg_works(..., languages = "en", only_text = TRUE, rights = c("Public domain in the USA.", "None"), distinct = TRUE, all_languages = FALSE, only_languages = TRUE)

Filter

glimpse(gutenberg_works()) lists all English language texts in project Gutenberg by whatever metafield you would like to filter on

gutenberg_works(author == "Dickens, Charles", languages = "en", only_text = TRUE,
    rights = NULL, distinct = TRUE, all_languages = FALSE, only_languages = TRUE)  #all english works regardless of copyright

gutenberg_id	title	author	gutenberg_author_id	language	gutenberg_bookshelf	rights	has_text
46	A Christmas Carol in Prose; Being a Ghost Story of Christmas	Dickens, Charles	37	en	Children’s Literature/Christmas/Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
98	A Tale of Two Cities	Dickens, Charles	37	en	Historical Fiction/Category: Historical Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
564	The Mystery of Edwin Drood	Dickens, Charles	37	en	Mystery Fiction/Category: Crime, Thrillers and Mystery/Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
580	The Pickwick Papers	Dickens, Charles	37	en	Best Books Ever Listings/Category: Humour/Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
588	Master Humphrey’s Clock	Dickens, Charles	37	en	Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
644	The Haunted Man and the Ghost’s Bargain	Dickens, Charles	37	en	Christmas/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
650	Pictures from Italy	Dickens, Charles	37	en	Category: Travel Writing/Category: British Literature	Public domain in the USA.	TRUE
653	The Chimes A Goblin Story of Some Bells That Rang an Old Year out and a New Year In	Dickens, Charles	37	en	Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
675	American Notes	Dickens, Charles	37	en	Category: Travel Writing/Category: British Literature	Public domain in the USA.	TRUE
676	The Battle of Life	Dickens, Charles	37	en	Christmas/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
678	The Cricket on the Hearth: A Fairy Tale of Home	Dickens, Charles	37	en	Children’s Literature/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
699	A Child’s History of England	Dickens, Charles	37	en	Children’s History/United Kingdom/Category: Children & Young Adult Reading/Category: History - British	Public domain in the USA.	TRUE
700	The Old Curiosity Shop	Dickens, Charles	37	en	Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
730	Oliver Twist	Dickens, Charles	37	en	Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
766	David Copperfield	Dickens, Charles	37	en	Harvard Classics/Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
786	Hard Times	Dickens, Charles	37	en	Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
807	Hunted Down: The Detective Stories of Charles Dickens	Dickens, Charles	37	en	Detective Fiction/Category: Short Stories/Category: Crime, Thrillers and Mystery/Category: British Literature	Public domain in the USA.	TRUE
809	Holiday Romance	Dickens, Charles	37	en	Children’s Picture Books/Category: Short Stories/Category: Children & Young Adult Reading/Category: Humour/Category: British Literature	Public domain in the USA.	TRUE
810	George Silverman’s Explanation	Dickens, Charles	37	en	Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
821	Dombey and Son	Dickens, Charles	37	en	Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
824	Speeches: Literary and Social	Dickens, Charles	37	en	Category: Essays, Letters & Speeches/Category: British Literature	Public domain in the USA.	TRUE
872	Reprinted Pieces	Dickens, Charles	37	en	Category: Short Stories/Category: Essays, Letters & Speeches/Category: Travel Writing/Category: British Literature	Public domain in the USA.	TRUE
882	Sketches by Boz, Illustrative of Every-Day Life and Every-Day People	Dickens, Charles	37	en	Category: Short Stories/Category: British Literature	Public domain in the USA.	TRUE
883	Our Mutual Friend	Dickens, Charles	37	en	Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
888	The Lazy Tour of Two Idle Apprentices	Dickens, Charles	37	en	Category: Humour/Category: Travel Writing/Category: British Literature	Public domain in the USA.	TRUE
912	Mudfog and Other Sketches	Dickens, Charles	37	en	Category: Short Stories/Category: Humour/Category: British Literature	Public domain in the USA.	TRUE
914	The Uncommercial Traveller	Dickens, Charles	37	en	Category: Essays, Letters & Speeches/Category: Travel Writing/Category: British Literature	Public domain in the USA.	TRUE
916	Sketches of Young Couples	Dickens, Charles	37	en	Category: Humour/Category: Essays, Letters & Speeches/Category: British Literature	Public domain in the USA.	TRUE
917	Barnaby Rudge: A Tale of the Riots of ’Eighty	Dickens, Charles	37	en	Historical Fiction/Category: Historical Novels/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
918	Sketches of Young Gentlemen	Dickens, Charles	37	en	Category: Humour/Category: British Literature	Public domain in the USA.	TRUE
922	Sunday Under Three Heads	Dickens, Charles	37	en	Category: Essays, Letters & Speeches/Category: British Literature	Public domain in the USA.	TRUE
924	To Be Read at Dusk	Dickens, Charles	37	en	Category: Short Stories/Category: Crime, Thrillers and Mystery/Category: British Literature	Public domain in the USA.	TRUE
927	The Lamplighter	Dickens, Charles	37	en	Category: Humour/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
963	Little Dorrit	Dickens, Charles	37	en	Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
967	Nicholas Nickleby	Dickens, Charles	37	en	Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
968	Martin Chuzzlewit	Dickens, Charles	37	en	Best Books Ever Listings/Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
1023	Bleak House	Dickens, Charles	37	en	Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
1289	Three Ghost Stories	Dickens, Charles	37	en	Category: Short Stories/Category: Humour/Category: British Literature	Public domain in the USA.	TRUE
1392	The Seven Poor Travellers	Dickens, Charles	37	en	Category: Short Stories/Category: British Literature	Public domain in the USA.	TRUE
1394	The Holly-Tree	Dickens, Charles	37	en	Category: Short Stories/Category: British Literature	Public domain in the USA.	TRUE
1400	Great Expectations	Dickens, Charles	37	en	Best Books Ever Listings/Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
1406	The Perils of Certain English Prisoners	Dickens, Charles	37	en	Category: Historical Novels/Category: Adventure/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
1407	A Message from the Sea	Dickens, Charles	37	en	Category: Short Stories/Category: British Literature	Public domain in the USA.	TRUE
1413	Tom Tiddler’s Ground	Dickens, Charles	37	en	Category: Short Stories/Category: British Literature	Public domain in the USA.	TRUE
1414	Somebody’s Luggage	Dickens, Charles	37	en	Category: Short Stories/Category: Humour/Category: British Literature	Public domain in the USA.	TRUE
1415	Doctor Marigold	Dickens, Charles	37	en	Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
1416	Mrs. Lirriper’s Lodgings	Dickens, Charles	37	en	Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
1419	Mugby Junction	Dickens, Charles	37	en	Category: Short Stories/Category: British Literature	Public domain in the USA.	TRUE
1421	Mrs. Lirriper’s Legacy	Dickens, Charles	37	en	Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
1422	Going into Society	Dickens, Charles	37	en	Category: Short Stories/Category: Humour/Category: British Literature	Public domain in the USA.	TRUE
1423	No Thoroughfare	Dickens, Charles	37	en	Category: Crime, Thrillers and Mystery/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
1435	Miscellaneous Papers	Dickens, Charles	37	en	Category: Essays, Letters & Speeches/Category: British Literature	Public domain in the USA.	TRUE
1465	The Wreck of the Golden Mary	Dickens, Charles	37	en	Category: Adventure/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
1467	Some Christmas Stories	Dickens, Charles	37	en	Christmas/Category: Short Stories/Category: British Literature	Public domain in the USA.	TRUE
2324	A House to Let	Dickens, Charles	37	en	Category: Short Stories/Category: Crime, Thrillers and Mystery/Category: British Literature	Public domain in the USA.	TRUE
9691	All the Year Round	Dickens, Charles	37	en		Copyrighted. Read the copyright notice inside this book for details.	TRUE
9696	A Christmas Carol	Dickens, Charles	37	en		Copyrighted. Read the copyright notice inside this book for details.	TRUE
9712	Hunted Down	Dickens, Charles	37	en	Detective Fiction	Copyrighted. Read the copyright notice inside this book for details.	TRUE
9729	Perils of Certain English Prisoners	Dickens, Charles	37	en		Copyrighted. Read the copyright notice inside this book for details.	TRUE
9732	Barnaby Rudge	Dickens, Charles	37	en	Historical Fiction	Copyrighted. Read the copyright notice inside this book for details.	TRUE
9733	Sketches by Boz, illustrative of everyday life and every-day people	Dickens, Charles	37	en		Copyrighted. Read the copyright notice inside this book for details.	TRUE
9736	Sunday under Three Heads	Dickens, Charles	37	en		Copyrighted. Read the copyright notice inside this book for details.	TRUE
9739	Cricket on the Hearth	Dickens, Charles	37	en	Children’s Literature	Copyrighted. Read the copyright notice inside this book for details.	TRUE
15618	The Loving Ballad of Lord Bateman	Dickens, Charles	37	en	Category: Poetry/Category: Humour/Category: Mythology, Legends & Folklore/Category: British Literature	Public domain in the USA.	TRUE
20795	The Cricket on the Hearth	Dickens, Charles	37	en	Children’s Literature/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
23344	The Magic Fishbone A Holiday Romance from the Pen of Miss Alice Rainbird, Aged 7	Dickens, Charles	37	en	Children’s Picture Books/Category: Children & Young Adult Reading/Category: Science-Fiction & Fantasy/Category: British Literature	Public domain in the USA.	TRUE
23452	The Trial of William Tinkling Written by Himself at the Age of 8 Years	Dickens, Charles	37	en	Children’s Picture Books/Category: Children & Young Adult Reading/Category: Humour/Category: Adventure/Category: British Literature	Public domain in the USA.	TRUE
23765	Captain Boldheart & the Latin-Grammar Master A Holiday Romance from the Pen of Lieut-Col. Robin Redforth, aged 9	Dickens, Charles	37	en	Children’s Picture Books/Category: Children & Young Adult Reading/Category: Humour/Category: Adventure/Category: British Literature	Public domain in the USA.	TRUE
25852	The Letters of Charles Dickens. Vol. 1, 1833-1856	Dickens, Charles	37	en	Category: Biographies/Category: Essays, Letters & Speeches/Category: British Literature	Public domain in the USA.	TRUE
25853	The Letters of Charles Dickens. Vol. 2, 1857-1870	Dickens, Charles	37	en	Category: Biographies/Category: Essays, Letters & Speeches/Category: British Literature	Public domain in the USA.	TRUE
25854	The Letters of Charles Dickens. Vol. 3, 1836-1870	Dickens, Charles	37	en	Category: Biographies/Category: Essays, Letters & Speeches/Category: British Literature	Public domain in the USA.	TRUE
25985	Bardell v. Pickwick	Dickens, Charles	37	en	Category: Humour/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
30127	Tales from Dickens	Dickens, Charles	37	en	Category: Short Stories/Category: British Literature	Public domain in the USA.	TRUE
30368	A Christmas Carol The original manuscript	Dickens, Charles	37	en	Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
32241	Dickens’ Stories About Children Every Child Can Read	Dickens, Charles	37	en	Category: Short Stories/Category: Children & Young Adult Reading/Category: British Literature	Public domain in the USA.	TRUE
35536	The Poems and Verses of Charles Dickens	Dickens, Charles	37	en	Category: Poetry/Category: British Literature	Public domain in the USA.	TRUE
37121	Charles Dickens’ Children Stories	Dickens, Charles	37	en	Category: Short Stories/Category: Children & Young Adult Reading/Category: British Literature	Public domain in the USA.	TRUE
40723	The Battle of Life: A Love Story	Dickens, Charles	37	en	Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
40729	“Old Scrooge”: A Christmas Carol in Five Staves. Dramatized from Charles Dickens’ Celebrated Christmas Story.	Dickens, Charles	37	en	Category: Plays/Films/Dramas/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
41739	A Christmas Carol; Or, The Miser’s Warning! (Adapted from Charles Dickens’ Celebrated Work.)	Dickens, Charles	37	en	Category: Plays/Films/Dramas/Category: British Literature	Public domain in the USA.	TRUE
41894	Christmas-Tide	Dickens, Charles	37	en	Category: Children & Young Adult Reading/Category: Essays, Letters & Speeches/Category: Parenthood & Family Relations/Category: Teaching & Education	Public domain in the USA.	TRUE
42232	A Child’s Dream of a Star	Dickens, Charles	37	en	Category: Short Stories/Category: British Literature	Public domain in the USA.	TRUE
43111	The Personal History of David Copperfield	Dickens, Charles	37	en	Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
43207	Scenes and Characters from the Works of Charles Dickens Being Eight Hundred and Sixty-six Pictures Printed from the Original Wood Blocks	Dickens, Charles	37	en	Category: British Literature/Category: Art	Public domain in the USA.	TRUE
46675	Oliver Twist; or, The Parish Boy’s Progress. Illustrated	Dickens, Charles	37	en	Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
47529	Oliver Twist, Vol. 1 (of 3)	Dickens, Charles	37	en	Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
47530	Oliver Twist, Vol. 2 (of 3)	Dickens, Charles	37	en	Category: Crime, Thrillers and Mystery/Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
47531	Oliver Twist, Vol. 3 (of 3)	Dickens, Charles	37	en	Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
47534	The Posthumous Papers of the Pickwick Club, v. 1 (of 2)	Dickens, Charles	37	en	Category: Humour/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
47535	The Posthumous Papers of the Pickwick Club, v. 2 (of 2)	Dickens, Charles	37	en	Category: Humour/Category: Novels/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
49125	Stories from Dickens	Dickens, Charles	37	en	Category: Short Stories/Category: Children & Young Adult Reading/Category: British Literature	Public domain in the USA.	TRUE
49683	Cruikshank’s Water Colours	Dickens, Charles	37	en	Category: British Literature/Category: Art	Public domain in the USA.	TRUE
49927	Pearl-Fishing; Choice Stories from Dickens’ Household Words; First Series	Dickens, Charles	37	en	Category: Short Stories/Category: British Literature	Public domain in the USA.	TRUE
50334	Pearl-Fishing; Choice Stories from Dickens’ Household Words; Second Series	Dickens, Charles	37	en	Category: Short Stories/Category: Classics of Literature/Category: British Literature	Public domain in the USA.	TRUE
52125	Nell and Her Grandfather, Told from Charles Dickens’s “The Old Curiosity Shop”	Dickens, Charles	37	en	Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
58157	Index of the Project Gutenberg Works of Charles Dickens	Dickens, Charles	37	en	Category: Encyclopedias/Dictionaries/Reference	Public domain in the USA.	TRUE
61193	Mr. Pickwick’s Christmas Being an Account of the Pickwickians’ Christmas at the Manor Farm, of the Adventures There; the Tale of the Goblin Who Stole a Sexton, and of the Famous Sports on the Ice	Dickens, Charles	37	en	Category: Humour/Category: Novels/Category: British Literature	Public domain in the USA.	TRUE
61668	Stories of Intellect	Dickens, Charles	37	en	Category: Short Stories/Category: British Literature	Public domain in the USA.	TRUE
71858	Holly berries from Dickens	Dickens, Charles	37	en	Category: Essays, Letters & Speeches/Category: British Literature	Public domain in the USA.	TRUE
75856	The children of Dickens	Dickens, Charles	37	en	Category: Essays, Letters & Speeches/Category: British Literature	Public domain in the USA.	TRUE

Search Titles

library of congress subject headings (lcsh) “Library of Congress Subject Headings (LCSH) is perhaps the most widely adopted subject indexing language in the world, has been translated into many languages, and is used around the world by libraries large and small” library of congress classification (lcc)< Link and find codes by subject.

gutenberg_subjects %>%
    filter(subject_type == "lcsh") %>%
    count(subject, sort = TRUE)
sci <- gutenberg_subjects %>%
    filter(subject_type == "lcc", subject == "Q")  #return a vectorized list of all texts about 'science' just outputting their ID
extract <- as.vector(sci$gutenberg_id)  #writes gutenberg ids to a numeric vector
print(extract)
gutenberg_works(gutenberg_id == extract)  #return in index of all english works regardless of copyright about the subject type Q (which is the code for science)

Download 1 Doc

Download by gutenberg_id – here are some other fields:
gutenberg_download(gutenberg_id, mirror = NULL, strip = TRUE, meta_fields = TRUE, verbose = TRUE, files = NULL)

great <- gutenberg_download(1400, meta_fields = c("title", "author"))  #strip dumps stuff from the header, includes author, title, text in the dataframe
head(great)

Download multiple documents

add meta fields

# more.dickens <- gutenberg_download(c(588,644,1400), meta_fields = c('title',
# 'author')) head(more.dickens)

Download all works by one author

That are in English and have full text. Write the downloaded files to a folder then comment out so it doesn’t download them anew every time you knit.

# alldickens <- gutenberg_works(author == 'Dickens, Charles')) #isolate dickens
# works dickids <- alldickens$gutenberg_id #extract vector of document ids
# alldickens <- gutenberg_download(dickids, meta_fields = c('title', 'author'))
# str(alldickens)

janeaustenr

read about the jane austen package here
Fulltexts for Jane Austen’s 6 completed novels:
Sense & Sensibility
Pride & Prejudice
Mansfield Park
Emma
Northanger Abbey
Persuasion

Take a peek

austen_books()  #all six texts in tidy format

text	book
SENSE AND SENSIBILITY	Sense & Sensibility
	Sense & Sensibility
by Jane Austen	Sense & Sensibility
	Sense & Sensibility
	Sense & Sensibility
	Sense & Sensibility
	Sense & Sensibility
	Sense & Sensibility
	Sense & Sensibility
CHAPTER 1	Sense & Sensibility
	Sense & Sensibility
	Sense & Sensibility
The family of Dashwood had long been settled in Sussex. Their estate	Sense & Sensibility
was large, and their residence was at Norland Park, in the centre of	Sense & Sensibility
their property, where, for many generations, they had lived in so	Sense & Sensibility
respectable a manner as to engage the general good opinion of their	Sense & Sensibility
surrounding acquaintance. The late owner of this estate was a single	Sense & Sensibility
man, who lived to a very advanced age, and who for many years of his	Sense & Sensibility
life, had a constant companion and housekeeper in his sister. But her	Sense & Sensibility
death, which happened ten years before his own, produced a great	Sense & Sensibility
alteration in his home; for to supply her loss, he invited and received	Sense & Sensibility
into his house the family of his nephew Mr. Henry Dashwood, the legal	Sense & Sensibility
inheritor of the Norland estate, and the person to whom he intended to	Sense & Sensibility
bequeath it. In the society of his nephew and niece, and their	Sense & Sensibility
children, the old Gentleman’s days were comfortably spent. His	Sense & Sensibility
attachment to them all increased. The constant attention of Mr. and	Sense & Sensibility
Mrs. Henry Dashwood to his wishes, which proceeded not merely from	Sense & Sensibility
interest, but from goodness of heart, gave him every degree of solid	Sense & Sensibility
comfort which his age could receive; and the cheerfulness of the	Sense & Sensibility
children added a relish to his existence.	Sense & Sensibility
	Sense & Sensibility
By a former marriage, Mr. Henry Dashwood had one son: by his present	Sense & Sensibility
lady, three daughters. The son, a steady respectable young man, was	Sense & Sensibility
amply provided for by the fortune of his mother, which had been large,	Sense & Sensibility
and half of which devolved on him on his coming of age. By his own	Sense & Sensibility
marriage, likewise, which happened soon afterwards, he added to his	Sense & Sensibility
wealth. To him therefore the succession to the Norland estate was not	Sense & Sensibility
so really important as to his sisters; for their fortune, independent	Sense & Sensibility
of what might arise to them from their father’s inheriting that	Sense & Sensibility
property, could be but small. Their mother had nothing, and their	Sense & Sensibility
father only seven thousand pounds in his own disposal; for the	Sense & Sensibility
remaining moiety of his first wife’s fortune was also secured to her	Sense & Sensibility
child, and he had only a life-interest in it.	Sense & Sensibility
	Sense & Sensibility
The old gentleman died: his will was read, and like almost every other	Sense & Sensibility
will, gave as much disappointment as pleasure. He was neither so	Sense & Sensibility
unjust, nor so ungrateful, as to leave his estate from his nephew;–but	Sense & Sensibility
he left it to him on such terms as destroyed half the value of the	Sense & Sensibility
bequest. Mr. Dashwood had wished for it more for the sake of his wife	Sense & Sensibility
and daughters than for himself or his son;–but to his son, and his	Sense & Sensibility
son’s son, a child of four years old, it was secured, in such a way, as	Sense & Sensibility
to leave to himself no power of providing for those who were most dear	Sense & Sensibility
to him, and who most needed a provision by any charge on the estate, or	Sense & Sensibility
by any sale of its valuable woods. The whole was tied up for the	Sense & Sensibility
benefit of this child, who, in occasional visits with his father and	Sense & Sensibility
mother at Norland, had so far gained on the affections of his uncle, by	Sense & Sensibility
such attractions as are by no means unusual in children of two or three	Sense & Sensibility
years old; an imperfect articulation, an earnest desire of having his	Sense & Sensibility
own way, many cunning tricks, and a great deal of noise, as to outweigh	Sense & Sensibility
all the value of all the attention which, for years, he had received	Sense & Sensibility
from his niece and her daughters. He meant not to be unkind, however,	Sense & Sensibility
and, as a mark of his affection for the three girls, he left them a	Sense & Sensibility
thousand pounds a-piece.	Sense & Sensibility
	Sense & Sensibility
Mr. Dashwood’s disappointment was, at first, severe; but his temper was	Sense & Sensibility
cheerful and sanguine; and he might reasonably hope to live many years,	Sense & Sensibility
and by living economically, lay by a considerable sum from the produce	Sense & Sensibility
of an estate already large, and capable of almost immediate	Sense & Sensibility
improvement. But the fortune, which had been so tardy in coming, was	Sense & Sensibility
his only one twelvemonth. He survived his uncle no longer; and ten	Sense & Sensibility
thousand pounds, including the late legacies, was all that remained for	Sense & Sensibility
his widow and daughters.	Sense & Sensibility
	Sense & Sensibility
His son was sent for as soon as his danger was known, and to him Mr.	Sense & Sensibility
Dashwood recommended, with all the strength and urgency which illness	Sense & Sensibility
could command, the interest of his mother-in-law and sisters.	Sense & Sensibility
	Sense & Sensibility
Mr. John Dashwood had not the strong feelings of the rest of the	Sense & Sensibility
family; but he was affected by a recommendation of such a nature at	Sense & Sensibility
such a time, and he promised to do every thing in his power to make	Sense & Sensibility
them comfortable. His father was rendered easy by such an assurance,	Sense & Sensibility
and Mr. John Dashwood had then leisure to consider how much there might	Sense & Sensibility
prudently be in his power to do for them.	Sense & Sensibility
	Sense & Sensibility
He was not an ill-disposed young man, unless to be rather cold hearted	Sense & Sensibility
and rather selfish is to be ill-disposed: but he was, in general, well	Sense & Sensibility
respected; for he conducted himself with propriety in the discharge of	Sense & Sensibility
his ordinary duties. Had he married a more amiable woman, he might	Sense & Sensibility
have been made still more respectable than he was:–he might even have	Sense & Sensibility
been made amiable himself; for he was very young when he married, and	Sense & Sensibility
very fond of his wife. But Mrs. John Dashwood was a strong caricature	Sense & Sensibility
of himself;–more narrow-minded and selfish.	Sense & Sensibility
	Sense & Sensibility
When he gave his promise to his father, he meditated within himself to	Sense & Sensibility
increase the fortunes of his sisters by the present of a thousand	Sense & Sensibility
pounds a-piece. He then really thought himself equal to it. The	Sense & Sensibility
prospect of four thousand a-year, in addition to his present income,	Sense & Sensibility
besides the remaining half of his own mother’s fortune, warmed his	Sense & Sensibility
heart, and made him feel capable of generosity.– “Yes, he would give	Sense & Sensibility
them three thousand pounds: it would be liberal and handsome! It would	Sense & Sensibility
be enough to make them completely easy. Three thousand pounds! he	Sense & Sensibility
could spare so considerable a sum with little inconvenience.”– He	Sense & Sensibility
thought of it all day long, and for many days successively, and he did	Sense & Sensibility
not repent.	Sense & Sensibility
	Sense & Sensibility
No sooner was his father’s funeral over, than Mrs. John Dashwood,	Sense & Sensibility
without sending any notice of her intention to her mother-in-law,	Sense & Sensibility
arrived with her child and their attendants. No one could dispute her	Sense & Sensibility
right to come; the house was her husband’s from the moment of his	Sense & Sensibility
father’s decease; but the indelicacy of her conduct was so much the	Sense & Sensibility
greater, and to a woman in Mrs. Dashwood’s situation, with only common	Sense & Sensibility
feelings, must have been highly unpleasing;–but in HER mind there was	Sense & Sensibility
a sense of honor so keen, a generosity so romantic, that any offence of	Sense & Sensibility
the kind, by whomsoever given or received, was to her a source of	Sense & Sensibility
immovable disgust. Mrs. John Dashwood had never been a favourite with	Sense & Sensibility
any of her husband’s family; but she had had no opportunity, till the	Sense & Sensibility
present, of shewing them with how little attention to the comfort of	Sense & Sensibility
other people she could act when occasion required it.	Sense & Sensibility
	Sense & Sensibility
So acutely did Mrs. Dashwood feel this ungracious behaviour, and so	Sense & Sensibility
earnestly did she despise her daughter-in-law for it, that, on the	Sense & Sensibility
arrival of the latter, she would have quitted the house for ever, had	Sense & Sensibility
not the entreaty of her eldest girl induced her first to reflect on the	Sense & Sensibility
propriety of going, and her own tender love for all her three children	Sense & Sensibility
determined her afterwards to stay, and for their sakes avoid a breach	Sense & Sensibility
with their brother.	Sense & Sensibility
	Sense & Sensibility
Elinor, this eldest daughter, whose advice was so effectual, possessed	Sense & Sensibility
a strength of understanding, and coolness of judgment, which qualified	Sense & Sensibility
her, though only nineteen, to be the counsellor of her mother, and	Sense & Sensibility
enabled her frequently to counteract, to the advantage of them all,	Sense & Sensibility
that eagerness of mind in Mrs. Dashwood which must generally have led	Sense & Sensibility
to imprudence. She had an excellent heart;–her disposition was	Sense & Sensibility
affectionate, and her feelings were strong; but she knew how to govern	Sense & Sensibility
them: it was a knowledge which her mother had yet to learn; and which	Sense & Sensibility
one of her sisters had resolved never to be taught.	Sense & Sensibility
	Sense & Sensibility
Marianne’s abilities were, in many respects, quite equal to Elinor’s.	Sense & Sensibility
She was sensible and clever; but eager in everything: her sorrows, her	Sense & Sensibility
joys, could have no moderation. She was generous, amiable,	Sense & Sensibility
interesting: she was everything but prudent. The resemblance between	Sense & Sensibility
her and her mother was strikingly great.	Sense & Sensibility
	Sense & Sensibility
Elinor saw, with concern, the excess of her sister’s sensibility; but	Sense & Sensibility
by Mrs. Dashwood it was valued and cherished. They encouraged each	Sense & Sensibility
other now in the violence of their affliction. The agony of grief	Sense & Sensibility
which overpowered them at first, was voluntarily renewed, was sought	Sense & Sensibility
for, was created again and again. They gave themselves up wholly to	Sense & Sensibility
their sorrow, seeking increase of wretchedness in every reflection that	Sense & Sensibility
could afford it, and resolved against ever admitting consolation in	Sense & Sensibility
future. Elinor, too, was deeply afflicted; but still she could	Sense & Sensibility
struggle, she could exert herself. She could consult with her brother,	Sense & Sensibility
could receive her sister-in-law on her arrival, and treat her with	Sense & Sensibility
proper attention; and could strive to rouse her mother to similar	Sense & Sensibility
exertion, and encourage her to similar forbearance.	Sense & Sensibility
	Sense & Sensibility
Margaret, the other sister, was a good-humored, well-disposed girl; but	Sense & Sensibility
as she had already imbibed a good deal of Marianne’s romance, without	Sense & Sensibility
having much of her sense, she did not, at thirteen, bid fair to equal	Sense & Sensibility
her sisters at a more advanced period of life.	Sense & Sensibility
	Sense & Sensibility
	Sense & Sensibility
	Sense & Sensibility
CHAPTER 2	Sense & Sensibility
	Sense & Sensibility
	Sense & Sensibility
Mrs. John Dashwood now installed herself mistress of Norland; and her	Sense & Sensibility
mother and sisters-in-law were degraded to the condition of visitors.	Sense & Sensibility
As such, however, they were treated by her with quiet civility; and by	Sense & Sensibility
her husband with as much kindness as he could feel towards anybody	Sense & Sensibility
beyond himself, his wife, and their child. He really pressed them,	Sense & Sensibility
with some earnestness, to consider Norland as their home; and, as no	Sense & Sensibility
plan appeared so eligible to Mrs. Dashwood as remaining there till she	Sense & Sensibility
could accommodate herself with a house in the neighbourhood, his	Sense & Sensibility
invitation was accepted.	Sense & Sensibility
	Sense & Sensibility
A continuance in a place where everything reminded her of former	Sense & Sensibility
delight, was exactly what suited her mind. In seasons of cheerfulness,	Sense & Sensibility
no temper could be more cheerful than hers, or possess, in a greater	Sense & Sensibility
degree, that sanguine expectation of happiness which is happiness	Sense & Sensibility
itself. But in sorrow she must be equally carried away by her fancy,	Sense & Sensibility
and as far beyond consolation as in pleasure she was beyond alloy.	Sense & Sensibility
	Sense & Sensibility
Mrs. John Dashwood did not at all approve of what her husband intended	Sense & Sensibility
to do for his sisters. To take three thousand pounds from the fortune	Sense & Sensibility
of their dear little boy would be impoverishing him to the most	Sense & Sensibility
dreadful degree. She begged him to think again on the subject. How	Sense & Sensibility
could he answer it to himself to rob his child, and his only child too,	Sense & Sensibility
of so large a sum? And what possible claim could the Miss Dashwoods,	Sense & Sensibility
who were related to him only by half blood, which she considered as no	Sense & Sensibility
relationship at all, have on his generosity to so large an amount. It	Sense & Sensibility
was very well known that no affection was ever supposed to exist	Sense & Sensibility
between the children of any man by different marriages; and why was he	Sense & Sensibility
to ruin himself, and their poor little Harry, by giving away all his	Sense & Sensibility
money to his half sisters?	Sense & Sensibility
	Sense & Sensibility
“It was my father’s last request to me,” replied her husband, “that I	Sense & Sensibility
should assist his widow and daughters.”	Sense & Sensibility
	Sense & Sensibility
“He did not know what he was talking of, I dare say; ten to one but he	Sense & Sensibility

Emma

cat(emma[1:50], sep = "\n")

## EMMA
## 
## By Jane Austen
## 
## 
## 
## 
## VOLUME I
## 
## 
## 
## CHAPTER I
## 
## 
## Emma Woodhouse, handsome, clever, and rich, with a comfortable home
## and happy disposition, seemed to unite some of the best blessings of
## existence; and had lived nearly twenty-one years in the world with very
## little to distress or vex her.
## 
## She was the youngest of the two daughters of a most affectionate,
## indulgent father; and had, in consequence of her sister's marriage, been
## mistress of his house from a very early period. Her mother had died
## too long ago for her to have more than an indistinct remembrance of
## her caresses; and her place had been supplied by an excellent woman as
## governess, who had fallen little short of a mother in affection.
## 
## Sixteen years had Miss Taylor been in Mr. Woodhouse's family, less as a
## governess than a friend, very fond of both daughters, but particularly
## of Emma. Between _them_ it was more the intimacy of sisters. Even before
## Miss Taylor had ceased to hold the nominal office of governess, the
## mildness of her temper had hardly allowed her to impose any restraint;
## and the shadow of authority being now long passed away, they had been
## living together as friend and friend very mutually attached, and Emma
## doing just what she liked; highly esteeming Miss Taylor's judgment, but
## directed chiefly by her own.
## 
## The real evils, indeed, of Emma's situation were the power of having
## rather too much her own way, and a disposition to think a little too
## well of herself; these were the disadvantages which threatened alloy to
## her many enjoyments. The danger, however, was at present so unperceived,
## that they did not by any means rank as misfortunes with her.
## 
## Sorrow came--a gentle sorrow--but not at all in the shape of any
## disagreeable consciousness.--Miss Taylor married. It was Miss Taylor's
## loss which first brought grief. It was on the wedding-day of this
## beloved friend that Emma first sat in mournful thought of any
## continuance. The wedding over, and the bride-people gone, her father and
## herself were left to dine together, with no prospect of a third to cheer
## a long evening. Her father composed himself to sleep after dinner, as
## usual, and she had then only to sit and think of what she had lost.

Homework

First, I will import the ‘sticks_messy.txt’ from my data folder. Then, I will write a simple cleaning function, which chains together a few cleaning commands (such as removing punctuation or various stop words). Then, I will split the cleaned version into one word per row. Next, I will determine how many tokens are in the file before and after cleaning (using ” ” as a separator). Finally, I will determine the TTR.

Import “sticks_messy.txt” from my data folder

sticks_messy <- readLines("data/sticks_messy.txt")
print(sticks_messy)

## [1] "Every 98 year 89769hlj Thanksgiving n@! ight O we flocked OIHJOIJ out behind : %^Dad as he KPOANL dragged the Santa suit to the road and draped it over a kind of crucifix he'd built out of metal pole in the yard. Super 352 %% Bowl week      the pole was dressed in a jersey and Rod's helmet and << > Rod had to clear it with Dad if he wanted ;! to take the helmet off. On the Fourth of July the pole was Uncle Sam, on Veteran’s Day a soldier,  on Halloween a ghost. The pole was Dad's only concession to       glee. We were allowed a single !)()_. Crayola from the box at a (& time. One Christmas Eve he shrieked at Kimmie for #@#%  wasting an apple slice. He hovered over us as we poured ketchup saying: good enough good enough good enough. Birthday parties ()(@ consisted of cupcakes, no ice cream. The first time I brought a date over she said: what's with Your dad and that pole? and I sat there blinking."                                                                                                                                                                                                                                                                                                                                                                                                                      
## [2] "We left h! +m++ mome, married,  had children of our own, found the seeds of meanness blooming also within us. Dad began dressing the pole with more >> mcomplexity and less discernible logic. He draped some kind of fur over it on Groundhog Day and lugged out a floodlight to ensure a shadow. When an earthquake struck Chile he lay the pole on its side and spray painted a rift in ++ the earth. Mom died and he dressed == _ the pole as Death and hung from the crossbar photos of Mom as a          baby. We'd stop by and find odd , M :? talismans from his youth arranged around the base: army medals, 17 theater tickets, old sweatshirts, tubes of Mom's makeup. One autumn he painted the pole bright yellow. He covered it with cotton swabs that winter for warmth and provided offspring by hammering in six crossed sticks around the yard. He ran lengths of string     between the pole and the sticks, and taped to the string letters of apology, admissions of error, pleas for understanding, all written in $  a frantic hand on index cards. He painted a sign saying < }{} LOVE and hung it from the pole and another that said FORGIVE? and then he died in the hall with the radio on and we  ?>? sold the house to a young couple who yanked out +++m | ? the pole and the sticks and #*1 left them by the road @ on garbage day."

Cleaning function

# create function with following cleaning: remove punctuation, remove the words
# 'a' and 'the', remove excess spaces, remove numbers
cleaning_func <- function(text) {
    x <- text
    x <- gsub("[[:punct:]]", " ", x)  #removes punctuation
    x <- gsub("\\ba\\b", " ", x)  #removes the word 'a'
    x <- gsub("\\bthe\\b", " ", x)  #removes the word 'the'
    x <- gsub(" {2,}", " ", x)  #removes excess spaces
    x <- gsub("[0-9]+", " ", x)  #removes digits 0-9
    return(x)
}

# test function
cleaning_func("This      is a test of the cleaning     function2!")

## [1] "This is test of cleaning function  "

# apply cleaning function to sticks_messy
sticks_cleaned <- cleaning_func(sticks_messy)
sticks_cleaned

## [1] "Every   year  hlj Thanksgiving n ight O we flocked OIHJOIJ out behind Dad as he KPOANL dragged Santa suit to road and draped it over kind of crucifix he d built out of metal pole in yard Super   Bowl week pole was dressed in jersey and Rod s helmet and Rod had to clear it with Dad if he wanted to take helmet off On Fourth of July pole was Uncle Sam on Veteran s Day soldier on Halloween ghost The pole was Dad s only concession to glee We were allowed single Crayola from box at time One Christmas Eve he shrieked at Kimmie for wasting an apple slice He hovered over us as we poured ketchup saying good enough good enough good enough Birthday parties consisted of cupcakes no ice cream The first time I brought date over she said what s with Your dad and that pole and I sat there blinking "                                                                                                                                                                                                                                                                                                                                                                
## [2] "We left h m mome married had children of our own found seeds of meanness blooming also within us Dad began dressing pole with more mcomplexity and less discernible logic He draped some kind of fur over it on Groundhog Day and lugged out floodlight to ensure shadow When an earthquake struck Chile he lay pole on its side and spray painted rift in earth Mom died and he dressed pole as Death and hung from crossbar photos of Mom as baby We d stop by and find odd M talismans from his youth arranged around base army medals   theater tickets old sweatshirts tubes of Mom s makeup One autumn he painted pole bright yellow He covered it with cotton swabs that winter for warmth and provided offspring by hammering in six crossed sticks around yard He ran lengths of string between pole and sticks and taped to string letters of apology admissions of error pleas for understanding all written in frantic hand on index cards He painted sign saying LOVE and hung it from pole and another that said FORGIVE and then he died in hall with radio on and we sold house to young couple who yanked out m pole and sticks and   left them by road on garbage day "

Tokenize the cleaned text - one word per row

# create data frame with sticks_cleaned and split to one word per row
sticks_cleaned_df <- data.frame(word = unlist(strsplit(sticks_cleaned, "\\s+")))

sticks_cleaned_df

x
Every
year
hlj
Thanksgiving
n
ight
O
we
flocked
OIHJOIJ
out
behind
Dad
as
he
KPOANL
dragged
Santa
suit
to
road
and
draped
it
over
kind
of
crucifix
he
d
built
out
of
metal
pole
in
yard
Super
Bowl
week
pole
was
dressed
in
jersey
and
Rod
s
helmet
and
Rod
had
to
clear
it
with
Dad
if
he
wanted
to
take
helmet
off
On
Fourth
of
July
pole
was
Uncle
Sam
on
Veteran
s
Day
soldier
on
Halloween
ghost
The
pole
was
Dad
s
only
concession
to
glee
We
were
allowed
single
Crayola
from
box
at
time
One
Christmas
Eve
he
shrieked
at
Kimmie
for
wasting
an
apple
slice
He
hovered
over
us
as
we
poured
ketchup
saying
good
enough
good
enough
good
enough
Birthday
parties
consisted
of
cupcakes
no
ice
cream
The
first
time
I
brought
date
over
she
said
what
s
with
Your
dad
and
that
pole
and
I
sat
there
blinking
We
left
h
m
mome
married
had
children
of
our
own
found
seeds
of
meanness
blooming
also
within
us
Dad
began
dressing
pole
with
more
mcomplexity
and
less
discernible
logic
He
draped
some
kind
of
fur
over
it
on
Groundhog
Day
and
lugged
out
floodlight

# 367 rows

Compare list size before and after cleaning

# Determine number of rows in tokenized messy list
sticks_messy_df <- data.frame(word = unlist(strsplit(sticks_messy, "\\s+")))

sticks_messy_df

x
Every
98
year
89769hlj
Thanksgiving
n@!
ight
O
we
flocked
OIHJOIJ
out
behind
:
%^Dad
as
he
KPOANL
dragged
the
Santa
suit
to
the
road
and
draped
it
over
a
kind
of
crucifix
he’d
built
out
of
metal
pole
in
the
yard.
Super
352
%%
Bowl
week
the
pole
was
dressed
in
a
jersey
and
Rod’s
helmet
and
<<
>
Rod
had
to
clear
it
with
Dad
if
he
wanted
;!
to
take
the
helmet
off.
On
the
Fourth
of
July
the
pole
was
Uncle
Sam,
on
Veteran’s
Day
a
soldier,
on
Halloween
a
ghost.
The
pole
was
Dad’s
only
concession
to
glee.
We
were
allowed
a
single
!)()_.
Crayola
from
the
box
at
a
(&
time.
One
Christmas
Eve
he
shrieked
at
Kimmie
for
#@#%
wasting
an
apple
slice.
He
hovered
over
us
as
we
poured
ketchup
saying:
good
enough
good
enough
good
enough.
Birthday
parties
()(@
consisted
of
cupcakes,
no
ice
cream.
The
first
time
I
brought
a
date
over
she
said:
what’s
with
Your
dad
and
that
pole?
and
I
sat
there
blinking.
We
left
h!
+m++
mome,
married,
had
children
of
our
own,
found
the
seeds
of
meanness
blooming
also
within
us.
Dad
began
dressing
the

nrow(sticks_messy_df)  #counts number of rows

## [1] 427

# 427 rows

# Compare to sticks_cleaned_df
sticks_cleaned_df

x
Every
year
hlj
Thanksgiving
n
ight
O
we
flocked
OIHJOIJ
out
behind
Dad
as
he
KPOANL
dragged
Santa
suit
to
road
and
draped
it
over
kind
of
crucifix
he
d
built
out
of
metal
pole
in
yard
Super
Bowl
week
pole
was
dressed
in
jersey
and
Rod
s
helmet
and
Rod
had
to
clear
it
with
Dad
if
he
wanted
to
take
helmet
off
On
Fourth
of
July
pole
was
Uncle
Sam
on
Veteran
s
Day
soldier
on
Halloween
ghost
The
pole
was
Dad
s
only
concession
to
glee
We
were
allowed
single
Crayola
from
box
at
time
One
Christmas
Eve
he
shrieked
at
Kimmie
for
wasting
an
apple
slice
He
hovered
over
us
as
we
poured
ketchup
saying
good
enough
good
enough
good
enough
Birthday
parties
consisted
of
cupcakes
no
ice
cream
The
first
time
I
brought
date
over
she
said
what
s
with
Your
dad
and
that
pole
and
I
sat
there
blinking
We
left
h
m
mome
married
had
children
of
our
own
found
seeds
of
meanness
blooming
also
within
us
Dad
began
dressing
pole
with
more
mcomplexity
and
less
discernible
logic
He
draped
some
kind
of
fur
over
it
on
Groundhog
Day
and
lugged
out
floodlight

nrow(sticks_cleaned_df)  #counts number of rows

## [1] 367

# 367 rows

Determine TTR of cleaned data frame

# Determine number of unique words in sticks_cleaned_df
sticks_unique <- length(unique(sticks_cleaned_df$word))
sticks_unique

## [1] 224

# 224

# Calculate TTR as unique / total
sticks_TTR <- sticks_unique/nrow(sticks_cleaned_df)
sticks_TTR

## [1] 0.6103542

# 0.6103542